Skip to content

Fix Cucumber test retries running under Suite#11850

Open
daniel-mohedano wants to merge 3 commits into
masterfrom
daniel.mohedano/fix-cucumber-retries
Open

Fix Cucumber test retries running under Suite#11850
daniel-mohedano wants to merge 3 commits into
masterfrom
daniel.mohedano/fix-cucumber-retries

Conversation

@daniel-mohedano

@daniel-mohedano daniel-mohedano commented Jul 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What Does This Do

  • Resolve the per-engine test-identifier factory in TestDataFactory by the leaf engine instead of the root one. Under @Suite, the unique id is rooted at the suite engine ([engine:junit-platform-suite]/[suite:…]/[engine:cucumber]/…). The previous approach, using UniqueId.getEngineId() returns the root engine, which in these cases would point to JUnit instead of Cucumber.

Motivation

Test execution policies (ATR, EFD, TIA, etc.) were silently not applied to Cucumber scenarios run via the JUnit Platform Suite Engine (@Suite + junit-platform-suite-engine).

The bug was masked because the instrumentation tests launch Cucumber directly via the Launcher API, with Cucumber as the root engine.

Additional Notes

Contributor Checklist

  • Format the title according to the contribution guidelines
  • Assign the type: and (comp: or inst:) labels in addition to any other useful labels
  • Avoid using close, fix, or any linking keywords when referencing an issue
    Use solves instead, and assign the PR milestone to the issue
  • Update the CODEOWNERS file on source file addition, migration, or deletion
  • Update public documentation with any new configuration flags or behaviors
  • Add your completed PR to the merge queue by commenting /merge. You can also:
    • Customize the commit message associated with the merge with /merge --commit-message "..."
    • Remove your PR from the merge queue with /merge -c
    • Skip all merge queue checks with /merge -f --reason "reason"; please use this judiciously, as some checks do not run at the PR-level (note: the PR still needs to be mergeable, this will only skip the pre-merge build)
    • Get more information in this doc

Jira ticket:

@daniel-mohedano daniel-mohedano added type: bug fix Bug fix comp: ci visibility Continuous Integration Visibility labels Jul 3, 2026
@daniel-mohedano daniel-mohedano changed the base branch from master to daniel.mohedano/jep-500-junit July 3, 2026 07:58
@daniel-mohedano daniel-mohedano changed the title Daniel.mohedano/fix cucumber retries Fix Cucumber test retries running under Suite Jul 3, 2026
@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 3, 2026 •
edited
Loading

Copy link
Copy Markdown

Test Environment - sbt-scalatest

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 53.12 54.33 $\color{green}{\blacktriangledown}$ -1.21 54.33 $\color{green}{\blacktriangledown}$ -1.21 56/210
agentEvpProxy 52.48 n/a n/a n/a n/a -

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 3, 2026 •
edited
Loading

Copy link
Copy Markdown

Test Environment - netflix-zuul

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 88.37 87.80 $\color{red}{\blacktriangle}$ +0.57 87.80 $\color{red}{\blacktriangle}$ +0.57 30/120
agentless 80.84 81.05 $\color{green}{\blacktriangledown}$ -0.21 81.05 $\color{green}{\blacktriangledown}$ -0.21 31/121
agentlessCodeCoverage 97.27 95.12 $\color{red}{\blacktriangle}$ +2.15 95.12 $\color{red}{\blacktriangle}$ +2.15 30/117
agentlessLineCoverage 112.71 111.62 $\color{red}{\blacktriangle}$ +1.09 111.62 $\color{red}{\blacktriangle}$ +1.09 29/117

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 3, 2026 •
edited
Loading

Copy link
Copy Markdown

Test Environment - nebula-release-plugin

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 37.85 36.42 $\color{red}{\blacktriangle}$ +1.43 36.42 $\color{red}{\blacktriangle}$ +1.43 28/104
agentless 36.18 36.42 $\color{green}{\blacktriangledown}$ -0.24 35.70 $\color{red}{\blacktriangle}$ +0.48 28/104
agentlessCodeCoverage 44.07 44.48 $\color{green}{\blacktriangledown}$ -0.41 44.48 $\color{green}{\blacktriangledown}$ -0.41 28/104
agentlessLineCoverage 75.50 74.82 $\color{red}{\blacktriangle}$ +0.68 74.82 $\color{red}{\blacktriangle}$ +0.68 27/103

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 3, 2026 •
edited
Loading

Copy link
Copy Markdown

Test Environment - pass4s

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 11.49 10.54 $\color{red}{\blacktriangle}$ +0.95 10.33 $\color{red}{\blacktriangle}$ +1.16 28/106
agentless 12.43 9.54 $\color{red}{\blacktriangle}$ +2.89 9.73 $\color{red}{\blacktriangle}$ +2.70 28/106
agentlessCodeCoverage 19.77 17.38 $\color{red}{\blacktriangle}$ +2.39 17.73 $\color{red}{\blacktriangle}$ +2.04 28/105

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 3, 2026 •
edited
Loading

Copy link
Copy Markdown

Test Environment - reactive-streams-jvm

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 21.17 22.09 $\color{green}{\blacktriangledown}$ -0.92 21.65 $\color{green}{\blacktriangledown}$ -0.48 28/119
agentless 17.57 18.82 $\color{green}{\blacktriangledown}$ -1.25 18.82 $\color{green}{\blacktriangledown}$ -1.25 28/117
agentlessCodeCoverage 18.57 19.99 $\color{green}{\blacktriangledown}$ -1.42 19.99 $\color{green}{\blacktriangledown}$ -1.42 26/115
agentlessLineCoverage 28.58 30.42 $\color{green}{\blacktriangledown}$ -1.84 29.82 $\color{green}{\blacktriangledown}$ -1.24 26/115

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 3, 2026 •
edited
Loading

Copy link
Copy Markdown

Test Environment - jolokia

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 92.75 93.23 $\color{green}{\blacktriangledown}$ -0.48 93.23 $\color{green}{\blacktriangledown}$ -0.48 29/111
agentless 88.04 89.58 $\color{green}{\blacktriangledown}$ -1.54 89.58 $\color{green}{\blacktriangledown}$ -1.54 28/109
agentlessCodeCoverage 98.22 99.00 $\color{green}{\blacktriangledown}$ -0.78 99.00 $\color{green}{\blacktriangledown}$ -0.78 28/107
agentlessLineCoverage 98.36 99.00 $\color{green}{\blacktriangledown}$ -0.64 99.00 $\color{green}{\blacktriangledown}$ -0.64 27/106

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 3, 2026 •
edited
Loading

Copy link
Copy Markdown

Test Environment - sonar-kotlin

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 12.00 12.62 $\color{green}{\blacktriangledown}$ -0.62 12.87 $\color{green}{\blacktriangledown}$ -0.87 28/107
agentless 13.28 12.62 $\color{red}{\blacktriangle}$ +0.66 12.12 $\color{red}{\blacktriangle}$ +1.16 28/106
agentlessCodeCoverage 15.82 15.11 $\color{red}{\blacktriangle}$ +0.71 15.11 $\color{red}{\blacktriangle}$ +0.71 28/106
agentlessLineCoverage 18.50 18.82 $\color{green}{\blacktriangledown}$ -0.32 19.20 $\color{green}{\blacktriangledown}$ -0.70 27/106

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 3, 2026 •
edited
Loading

Copy link
Copy Markdown

Test Environment - okhttp

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 21.62 19.20 $\color{red}{\blacktriangle}$ +2.42 19.20 $\color{red}{\blacktriangle}$ +2.42 30/115
agentless 20.80 19.20 $\color{red}{\blacktriangle}$ +1.60 19.20 $\color{red}{\blacktriangle}$ +1.60 29/113
agentlessCodeCoverage 23.06 22.54 $\color{red}{\blacktriangle}$ +0.52 22.09 $\color{red}{\blacktriangle}$ +0.97 29/111
agentlessLineCoverage 45.57 44.48 $\color{red}{\blacktriangle}$ +1.09 44.48 $\color{red}{\blacktriangle}$ +1.09 28/116

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 3, 2026 •
edited
Loading

Copy link
Copy Markdown

Test Environment - spring_boot

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 16.36 16.36 $\color{gray}{\bullet}$ 0.00 16.04 $\color{red}{\blacktriangle}$ +0.32 30/103
agentless 9.68 9.73 $\color{green}{\blacktriangledown}$ -0.05 9.73 $\color{green}{\blacktriangledown}$ -0.05 31/104
agentlessCodeCoverage 14.04 13.40 $\color{red}{\blacktriangle}$ +0.64 13.40 $\color{red}{\blacktriangle}$ +0.64 30/102
agentlessLineCoverage 33.82 32.95 $\color{red}{\blacktriangle}$ +0.87 32.30 $\color{red}{\blacktriangle}$ +1.52 29/102

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 3, 2026 •
edited
Loading

Copy link
Copy Markdown

Test Environment - sonar-java

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 34.45 17.73 $\color{red}{\blacktriangle}$ +16.72 16.04 $\color{red}{\blacktriangle}$ +18.41 28/110
agentless 11.70 17.38 $\color{green}{\blacktriangledown}$ -5.68 19.59 $\color{green}{\blacktriangledown}$ -7.89 28/109
agentlessCodeCoverage 69.89 87.80 $\color{green}{\blacktriangledown}$ -17.91 87.80 $\color{green}{\blacktriangledown}$ -17.91 28/109
agentlessLineCoverage 156.96 147.69 $\color{red}{\blacktriangle}$ +9.27 144.77 $\color{red}{\blacktriangle}$ +12.19 27/108

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

@daniel-mohedano daniel-mohedano marked this pull request as ready for review July 3, 2026 09:38
@daniel-mohedano daniel-mohedano requested a review from a team as a code owner July 3, 2026 09:38
datadog-datadog-prod-us1-2[bot]
datadog-datadog-prod-us1-2 Bot reviewed Jul 3, 2026
View reviewed changes

@datadog-datadog-prod-us1-2 datadog-datadog-prod-us1-2 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: PASS

More details

The fix correctly uses JUnitPlatformUtils.getEngineId() — which walks the UniqueId segment list backward to return the innermost engine-type segment — instead of UniqueId.getEngineId(), which returns the root segment. Validated against 11 UniqueId shapes including all production patterns (plain Cucumber, Suite+Cucumber, Suite+Jupiter, Suite+Spock, no-engine, container variants); every case dispatches to the correct factory. No regressions found.

Was this helpful? React 👍 or 👎

📊 Validated against 11 scenarios · Open Bits AI session

🤖 Datadog Autotest · Commit 70060ee · What is Autotest? · Any feedback? Reach out in #autotest

Base automatically changed from daniel.mohedano/jep-500-junit to master July 9, 2026 11:32
@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 69.73% (+12.79%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: b2d4130 | Docs | Datadog PR Page | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.02 s 13.89 s [+0.1%; +1.8%] (maybe worse)
startup:insecure-bank:tracing:Agent 12.93 s 12.99 s [-1.3%; +0.3%] (no difference)
startup:petclinic:appsec:Agent 16.87 s 16.26 s [-0.7%; +8.2%] (no difference)
startup:petclinic:iast:Agent 16.32 s 16.81 s [-7.2%; +1.3%] (no difference)
startup:petclinic:profiling:Agent 16.71 s 16.60 s [-0.4%; +1.8%] (no difference)
startup:petclinic:sca:Agent 16.84 s 16.67 s [-0.1%; +2.1%] (no difference)
startup:petclinic:tracing:Agent 16.07 s 15.94 s [-0.2%; +1.8%] (no difference)

Commit: b2d4130e · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Jul 9, 2026

Copy link
Copy Markdown

Test Environment - heliboard

Job Status: 🟢 success

Scenario This PR (%) 7d median Δ 7d 30d median Δ 30d runs (7d/30d)
agent 10.36 9.54 $\color{red}{\blacktriangle}$ +0.82 9.54 $\color{red}{\blacktriangle}$ +0.82 11/11

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@datadog-datadog-prod-us1-2 datadog-datadog-prod-us1-2[bot] datadog-datadog-prod-us1-2[bot] left review comments
@anmarchenko anmarchenko anmarchenko approved these changes

Assignees

No one assigned

Labels

comp: ci visibility Continuous Integration Visibility type: bug fix Bug fix

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@daniel-mohedano @anmarchenko